Add launch_custom subprocess dispatcher for CustomResourceConfig#626
Queued
kmontemayor2-sc wants to merge 13 commits into
Queued
Add launch_custom subprocess dispatcher for CustomResourceConfig#626kmontemayor2-sc wants to merge 13 commits into
kmontemayor2-sc wants to merge 13 commits into
Conversation
Introduces a new `oneof` arm on `TrainerResourceConfig` / `InferencerResourceConfig` that lets callers describe a launcher as a shell command + positional args, instead of a fixed-shape Vertex AI / KFP / local resource config. The proto carries no semantics here — the dispatcher is added in a follow-up PR; this commit only ships the message, regenerated bindings, and the wrapper-property update so downstream code can read `wrapper.trainer_config` and get a `CustomResourceConfig` back. The diff includes a long tail of cosmetic Scala changes outside `gigl_resource_config/` because scalapbc regenerates every sibling proto's emitted source whenever any one proto in the same directory changes. Reviewers can scope to `CustomResourceConfig.scala` and the `*ResourceConfig.scala` siblings that gain the new oneof case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
/all_test |
Contributor
GiGL Automation@ 23:08:16UTC : 🔄 @ 23:12:40UTC : ❌ Workflow failed. |
Contributor
GiGL Automation@ 23:08:18UTC : 🔄 @ 23:14:32UTC : ❌ Workflow failed. |
Contributor
GiGL Automation@ 23:08:18UTC : 🔄 @ 23:18:08UTC : ✅ Workflow completed successfully. |
Contributor
GiGL Automation@ 23:08:22UTC : 🔄 @ 23:10:11UTC : ✅ Workflow completed successfully. |
Contributor
GiGL Automation@ 23:08:24UTC : 🔄 @ 24:38:24UTC : ✅ Workflow completed successfully. |
Contributor
GiGL Automation@ 23:08:25UTC : 🔄 @ 24:24:39UTC : ✅ Workflow completed successfully. |
357c3cc to
2ef62bd
Compare
CustomResourceConfig is launcher-pluggable — there is no concrete machine spec to validate against (no machine_type, num_workers, GPU config, etc.). The wrapper's trainer_config / inferencer_config properties now return a union that includes CustomResourceConfig (introduced earlier in this PR), but _validate_machine_config does not accept it. Add an isinstance early-return guard at each call site that logs the skip and returns. Shape and backend-compatibility validation for CustomResourceConfig come in a follow-up PR; this commit only makes the existing validation flow type-clean against the widened union. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2ef62bd to
837dbc5
Compare
Implements `launch_custom`, a thin shim that takes a populated
`CustomResourceConfig` and shells out via
`subprocess.run(shell=True, check=True)`. The proto's `command` is
a shell snippet (so leading `KEY=VALUE` env assignments parse
naturally) and `args[]` are individually `shlex.quote`-d before
joining, so values containing whitespace survive the shell pass.
The dispatcher performs no template substitution: `command` and
`args[]` are taken verbatim, and any placeholder text reaches
`subprocess.run` literally. Consumers that want runtime-context
substitution (e.g. ${gigl:foo}) should resolve it at YAML-load
time before the proto reaches this module. No call site in the
rest of the repo invokes `launch_custom` yet — wiring is added in
a follow-up PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
837dbc5 to
066c473
Compare
…b.com/Snapchat/GiGL into kmonte/custom-resource-config-pr1-proto
The proto message describes a launcher command (command + args), not a resource shape (machines / replicas / pools) like its sibling oneof arms (VertexAiResourceConfig, KFPResourceConfig, LocalResourceConfig). Aligning the name with the supporting Python (custom_launcher.py / launch_custom) removes the misleading "Resource" suffix. Field names on TrainerResourceConfig / InferencerResourceConfig stay (custom_trainer_config / custom_inferencer_config) so the sibling field-name pattern is preserved. Regenerated pb2 / pyi / Scala bindings. Wrapper + validator + test references updated.
…stom-resource-config-pr2-launcher
Follow-up to the proto rename. Updates the launcher module's import, parameter type annotation, docstring, and error message, plus its test file. Also renames the launcher parameter from `custom_resource_config` to `custom_launcher_config` to match the new proto type name.
svij-sc
approved these changes
May 13, 2026
Any commits made after this event will not be merged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
launch_custom, a thin shim that takes a populatedCustomResourceConfigand shells out viasubprocess.run(shell=True, check=True). The proto'scommandisa shell snippet (so leading
KEY=VALUEenv assignments parsenaturally) and
args[]are individuallyshlex.quote-d beforejoining, so values containing whitespace survive the shell pass.
The dispatcher performs no template substitution:
commandandargs[]are taken verbatim, and any placeholder text reachessubprocess.runliterally. Consumers that want runtime-contextsubstitution (e.g. ${gigl:foo}) should resolve it at YAML-load
time before the proto reaches this module. No call site in the
rest of the repo invokes
launch_customyet — wiring is added ina follow-up PR.